Skip to content

refactor(http-clients)!: Replace crawl() and stream() with send_request - #2349

Merged
vdusek merged 4 commits into
apify:v2from
Mantisus:refactor-http-client
Oct 30, 2026
Merged

vdusek merged 4 commits into
apify:v2from
Mantisus:refactor-http-client

Conversation

@Mantisus

@Mantisus Mantisus commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Description

HttpClient now has a single public method, send_request, which returns the response before its body is read. A client implements only _fetch, which sends one request. The base class follows redirects, sends and stores the session cookies on every hop, and drops credentials when a redirect leaves the origin, the same way for every client. crawl() and stream() are removed.

The timeout of send_request covers the redirect chain and reading the body, every client takes follow_redirects, and HttpResponse is an abstract base class that closes the response when used as an async context manager. The v2 upgrade guide describes what changes for existing code and custom clients.

Issues

@Mantisus Mantisus self-assigned this Oct 22, 2026
@github-actions github-actions Bot added the t-tooling Issues with this label are in the ownership of the tooling team. label Oct 22, 2026
@Mantisus Mantisus changed the title refactor{http-clients}: Refactor HttpClient refactor(http-clients)!: Refactor HttpClient Oct 22, 2026
@Mantisus
Mantisus force-pushed the refactor-http-client branch 2 times, most recently from cb8b05a to c20c567 Compare October 24, 2026 16:22
@Mantisus
Mantisus marked this pull request as ready for review October 24, 2026 16:22
@Mantisus
Mantisus requested review from Pijukatel and vdusek and removed request for vdusek October 24, 2026 16:22
@Mantisus Mantisus changed the title refactor(http-clients)!: Refactor HttpClient refactor(http-clients)!: Move redirect and cookie handling into a shared HttpClient base class Oct 24, 2026
@Mantisus
Mantisus force-pushed the refactor-http-client branch from c20c567 to 887e251 Compare October 25, 2026 16:45
@vdusek vdusek changed the title refactor(http-clients)!: Move redirect and cookie handling into a shared HttpClient base class refactor(http-clients)!: Handle redirects and cookies in the HttpClient base class and drop crawl() Oct 28, 2026

@vdusek vdusek left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a few minor comments, otherwise LGTM

Comment thread tests/unit/_utils/test_http.py Resolved
Comment thread tests/unit/_utils/test_http.py Resolved
@Mantisus
Mantisus force-pushed the refactor-http-client branch 2 times, most recently from 8d72acd to 38558a1 Compare October 29, 2026 14:39
@vdusek vdusek linked an issue Oct 30, 2026 that may be closed by this pull request
@vdusek vdusek changed the title refactor(http-clients)!: Handle redirects and cookies in the HttpClient base class and drop crawl() refactor(http-clients)!: Reduce HttpClient to send_request and move redirects and cookies to the base class Oct 30, 2026
@vdusek vdusek changed the title refactor(http-clients)!: Reduce HttpClient to send_request and move redirects and cookies to the base class refactor(http-clients)!: Replace crawl() and stream() with send_request Oct 30, 2026
@vdusek
vdusek merged commit 9aba398 into apify:v2 Oct 30, 2026
35 checks passed
vdusek added a commit that referenced this pull request Oct 31, 2026
…on reuse (#2373)

Two tests added in #2349 fail intermittently in CI (seen on #2371).

- `test_closing_unread_response_reuses_connection[impit]`: impit returns
a drained connection to its pool in the background, so the request sent
right after a close sometimes opens a second connection. Under load this
failed in 2 of 120 local runs. The test now allows up to two
connections. A working client opens 1 or 2, and one that drops the
connections of closed responses always opens 3 (300 runs each), so the
test still catches that regression.
- `test_read_again_after_timeout[httpx]`: the 200 ms timeout covered the
whole request, and the slow macOS runner didn't get the headers in time.
The test now uses a 10 s timeout and moves the event loop clock past the
deadline before the second `read()`, so it doesn't wait on the wall
clock.

Both tests still fail when the behavior they cover is broken. After the
change, 1,440 runs under load passed.

*✍️ Drafted by Claude Code*
vdusek pushed a commit that referenced this pull request Nov 3, 2026
…HttpClient.crawl` (#2367)

### Description

`HttpCrawlingResult` is removed. Only `HttpClient.crawl` returned it,
and #2349 removes `crawl`. `HttpCrawlingContext` used to inherit
`http_response` from it and now declares the field itself, so the
crawling contexts work as before. The v2 upgrade guide notes the
removal.

### Issues

- Follow-up: #2349

*✍️ Drafted by Claude Code*
vdusek added a commit that referenced this pull request Nov 3, 2026
### Description

A `Cookie` header passed with a request or set on the client is now
merged with the session cookies, with every HTTP client and in
`context.send_request` of `PlaywrightCrawler`. A cookie in the header
wins over a session cookie of the same name and isn't stored in the
session. An empty `Cookie` header no longer keeps the session cookies
out. The v2 upgrade guide describes the change.

### Issues

- Part of: #2307
- Follow-up: #2349

*✍️ Drafted by Claude Code*

Co-authored-by: Vlada Dusek <v.dusek96@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reconsider HttpClient interface

2 participants